Aeroacoustic post - processing with MapReduce

Present day large-scale computational fluid dynamics simulations can easily produce tens, if not hundreds, of terabytes of useful data. While computational capacity continues to increase according to Moore’s law, the speed of input-output (I/O) to data storage systems has not increased at the same rate. This means that the gap between processing speed and bandwidth to storage systems is increasing exponentially. This trend is in part fueled by the fact that supercomputer power is most often measured in floating point operations per second (FLOPS), while other metrics receive less attention. If the gap between processing and storage speed continues to grow, it will drive scientific data processing towards an in-situ paradigm in which very little data are ever stored. Instead, “post-processing” routines will need to be performed “on-the-fly,” in tandem with the simulations that they analyze. If a different data analysis is desired in the future, it will require re-running the simulation. Note that this paradigm may be at odds with usual scientific procedure where data are collected and then analyzed in a progressive fashion. Understanding one aspect of the data naturally leads to a host of additional questions. In other words, scientific data sets often contain unexpected effects that cannot be predicted ahead of time. The purpose of this brief, therefore, is to explore new technologies enabling fast access to large quantities of stored data as an alternative to the in-situ paradigm. In particular, we look to data-access techniques developed for web search engines like Google, which must constantly query enormous databases pertaining to the state of the Internet. Such databases are too large to be stored on any one disk – instead they reside on thousands or tens of thousands of disks. The solution to fast access lies in expressing a query in a special format known as “MapReduce,” which is in itself a programming paradigm. If a postprocessing task is expressible in this fashion, MapReduce enables the code implementing the task (map phase) to be sent directly to the data residing on distributed disks, rather than requiring the data to be sent to the code. Only a small amount of data representing the desired result is communicated at the end (reduce phase). In this way, effective I/O throughput may be dramatically increased.